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Abstract 

PISA, the OECD’s international program of assessment of reading, 
scientific and mathematical literacy (www.oecd.org/pisa), aims to 
assess the ability of 15 year olds to use the knowledge and skills that 
have acquired at school in real world tasks and challenges. It also uses 
questionnaires to gather data on students’ attitudes to learning and the 
conditions of schooling. Since 2000, PISA has tested the mathematical 
literacy of 15 year old students from many countries around the world. 
This paper describes the design of the PISA assessments, discusses 
mathematical literacy and reports on a selection of results from the 
PISA assessments, associated surveys and related analyses to give a 
flavour of the information that has resulted from this major 
international initiative. Results for Indonesia are compared with the 
OECD average and with a selection of countries, addressing issues of 
overall achievement, equity, and classroom environment. 


Abstrak 

PISA, adalah program internasional OECD untuk mengevaluasi 
kemampuan membaca, sains dan matematika (lihat 
www.oecd.org/pisa), bertujuan untuk mengetahui kemampuan anak 
usia 15 tahun dalam menggunakan kemampuan dan keahlian yang telah 
mereka pelajari di sekolah dalam menjalani kehidupan mereka sehari- 
hari di zaman global yang penuh tantangan. Program ini juga 
menggunakan angket untuk mendapatkan informasi terkait sikap siswa 
untuk belajar serta kondisi sekolah. Sejak tahun 2000, telah mengases 
kemampuan matematika siswa usia 15 tahun dari berbagai negara di 
dunia. Paper ini menjelaskan bagaimana soal PISA dibuat, 
mendiskusikan literasi matematika dan melaporkan hasil PISA untuk 
beberapa negara tertentu, hasil analisisnya untuk memberikan 
pemahaman mendalam yang telah dihasilkan dari program 
internasional ini. Hasil siswa Indonesia dibandingkan dengan rata-rata 
negara OECD serta beberapa negara yang dipilih, terkait pemahaman 
secara umum, kebersamaan dan lingkungan kelas. 


Introduction to PISA 

PISA is the acronym for the ‘Programme for International Student Assessment’, 
organised by the OECD in conjunction with a group of other participating countries, 
including Indonesia. The first survey took place in 2000, and then every 3 years since 
that time. PISA measures knowledge and skills of 15-year-olds, an age at which 
students in most countries are nearing the end of compulsory schooling. The focus is 
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on areas that are important for life after school, including mathematics. PISA is a 
statistically rigorous programme to assess student performance and to collect data on 
the student, family and institutional factors that can help to explain differences in 
performance in countries around the world. Substantial efforts and resources are 
devoted to achieving cultural and linguistic breadth and balance in the assessment 
materials. Stringent quality assurance mechanisms are applied in translation, sampling 
and data collection to ensure that the results are as meaningful as possible. The aim is 
to significantly improve understanding of the outcomes of education in the OECD 
countries, as well as in a growing number of countries at earlier stages of economic 
development who are choosing to participate. 

The OECD identifies the key features of PISA as: 

• policy orientation, with the major aim of informing educational policy and 
practice; 

• the PISA concept of “literacy” (see below) with a foundation of assessment of 
literacy for reading, mathematics and science; 

• its relevance to lifelong learning, so that assessment of knowledge is 
supplemented by reports on motivation to learn, attitudes towards learning and 
learning strategies; 

• its regularity, enabling countries to monitor improvements in educational 
outcomes in the light of other countries’ performances on assessments held 
every 3 years; 

• measurement of student performance alongside characteristics of students and 
schools, in order to explore some of the main features associated with 
educational success; 

• breadth, with over 60 countries and economies participating by 2009, 
representing around 90% of the world economy. 

PISA is one of two major international assessments of mathematics and science 
achievement, the other being TIMSS (http://timss.bc.edu/), which has assessed Grade 
4 and 8 students (and sometimes Grade 12) regularly since 1994/1995. A major 
difference is that TIMSS aims to make an assessment of the (common aspects of the) 
mathematics curriculum as taught in participating countries, whereas PISA starts from 
concern for the mathematical literacy (see below) that is judged to be valuable to 15- 
year-olds for their lives. 
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Four PISA surveys have taken place so far, in 2000, 2003, 2006 and 2009, focusing 
on reading, mathematics literacy and science literacy and this three year pattern will 
continue. In each assessment year, one of these three ‘literacies’ is the major focus. 
The cycle began with reading in 2000, mathematics in 2003 and science in 2006. The 
2009 assessment represented a new round, with reading again the major focus. PISA 
2012 will focus on mathematics as the main domain as in 2003. In each assessment, 
trend data is collected through an abbreviated test on the other two domains, using 
‘link items’ from the earlier assessment to give results that can indicate trends. 
Students also answer a 30 minute background questionnaire, providing information 
about themselves, their attitudes to learning and their homes. School principals answer 
a questionnaire about their schools. These questionnaires provide baseline information 
about the conditions of schooling in different countries, and enable the examination of 
issues such as equity of schooling and effective practices. 

PISA is also developing other assessments. For example, it measures Information and 
Communication Technology skills and assesses the reading of electronic texts. In 
2012, there will be an optional component on financial literacy. Since the science 
assessment of 2006, computer-based assessments have also been used to support a 
wider range of dynamic and interactive tasks. The mathematics assessment for 2012 
will have an optional computer-administered component, which will provide new 
opportunities for presentations of items and may also test some aspects of doing 
mathematics assisted by a computer. The 30th meeting of the PISA Governing Board 
in November 2010 proposed “moving PISA from a paper-based assessment towards a 
technology-rich assessment in 2015 as well as from traditional item formats to the 
kind of innovative assessment formats which computer-delivery would enable (OECD 
2010b). PISA has also conducted tests of general problem solving, and will do so 
again in 2012. The problem solving assessment taps students’ “capacity to use 
cognitive processes to confront and resolve real, cross-disciplinary situations where 
the solution path is not immediately obvious and where the literacy domains or 
curricular areas that might be applicable are not within a single domain of 
mathematics, science or reading” (OECD 2003b, p. 156). The mathematics 
assessment also contains many items that might be considered problem solving, but 
they draw explicitly on mathematics content. 
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Participating Countries and Students 

All OECD countries have participated in PISA since its inception in 2000. As a 
partner country Indonesia has also participated since 2000. There is growing 
participation by Asian countries in PISA. Asian countries have been prominent for 
over a decade in the TIMSS assessments, with public attention attracted by some very 
high Asian performances. By 2009, the Asian countries participating in PISA were the 
OECD countries Australia, Japan and Korea and the partner countries Indonesia, three 
parts of China which are considered as separate economies (Hong Kong, Shanghai 
and Macao), Chinese Taipei, Malaysia, Singapore, and Thailand. Two parts of India 
(Himachal Pradesh, Tamil Nadu) and Viet Nam participated in the survey late. In 
2009, there were 65 participating countries made up of 34 OECD countries and 31 
partner countries and a further 9 completing late. 

Schools in each country are randomly selected by the international contracting 
consortium for participation. At these schools, the test is given to all students who are 
between age 15 years 3 months and age 16 years 2 months at the time of the test, 
rather than to students in a specific year of school as with TIMSS. The selection of 
schools and students is kept as inclusive as possible, so that the sample of students 
comes from a broad range of backgrounds and abilities. The sampling is carried out 
very strictly. Sometimes the results of countries that participated in the surveys are not 
included in all of the analyses, because they failed to meet the strict sampling criteria. 
For example, even though PISA tests were conducted in the Netherlands in 2000, the 
response rate of schools was below that required for inclusion and thus the 
Netherlands is excluded from trend analysis beginning with PISA 2000. 

There are many ways in which the design of the sample impacts on the results, and 
these need to be considered, especially when comparing results from different 
countries, or when comparing results from TIMSS and PISA. PISA selects students 
by age, rather than sampling whole classes within grade levels as is used by TIMSS. 
Hence a survey of teachers is not as appropriate for PISA as it is for TIMSS, although 
a survey of the school is informative, especially in locations where 15-year-old 
students have usually attended that school for some time. In this circumstance, the 
outcomes of student learning could be expected to have been strongly influenced by 
actions taken at that school, and so the school questionnaire should reveal information 
about conditions to teach mathematical literacy well. Both age and number of years of 
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schooling affect performance of students and so the difference in sampling methods of 
TIMSS and PISA is one reason for the moderate differences in the rankings that they 
produce. This is pertinent when comparing the results on TIMSS and PISA for the 
Australian states, for example. In some states, 15-year-old students have spent more 
years at school than in other states and hence are probably in a higher grade level. 
These factors need to be considered and accounted for, before concluding that there 
are important differences in the knowledge and skills attained in different school 
systems (Stacey & Stephens, 2008). School policies about progression also affect the 
sampling. In some countries, almost all students of a given age are in the same grade, 
but in others, including Indonesia, nearly half of the 15-year-olds are not in the modal 
grade. (OECD 2010a, Vol IV, Table IV.3.1). 

Surveys, Questionnaires, and Reporting 

PISA surveys are made up of both multiple-choice questions and constructed response 
questions. Each PISA survey includes about seven hours of test material but 
individual students each do a two-hour subset of this. Country and other group results 
are constructed by combining these results. Since different students do items of 
potentially different difficulty, the results of individual students on PISA tests are not 
meaningful. 

In mathematics and science, the test items are grouped into units which start by 
describing a real world situation or problem. Mathematics units usually begin with a 
description of a situation that might be encountered in real life in words and pictures 
and possibly symbols (e.g. a map for a journey, authentic tables of data, plans for a 
house), a formula to calculate something practical) and a series of questions requiring 
students to use this information, for example, to calculate quantities and interpret 
results, with all aspects having face validity as sensible ways to use mathematics. 
RISING CRIMES, a sample mathematics unit (OECD 2006, p. 94) begins with a very 
striking stylised graph of reported crimes per 100 000 inhabitants of a town. For item 
1, students have to read a point from the graph. For item 2, a second graph of the same 
information is given, supposedly produced by manufacturers of burglar alarms. 
Students have to identify features such as how the graph has been altered to convey a 
message. A non-linear scale was used. They then have to construct a graph which the 
police could use to show from the same data that crime has recently decreased. This 
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unit is about interpretation of information provided by graphs and it also requires 
students to analyse how the visual appearance of the graph can be manipulated (e.g. 
by graphing only a subset of the range) to emphasise a particular message. 

This type of item illustrates ‘mathematical literacy’ since it is highly relevant to the 
way in which citizens in all societies need to be able to deal with information 
presented in the media. Another unit, STUDENT HEIGHTS (OECD 2006, p. 104), 
gives information about the average height of students in a class, and the ranges and 
averages for boys and girls both when the whole class is present and when two 
students are absent. Students have to evaluate the truth of several statements about 
who was absent. A sample science item (OECD 2006, p. 28) begins with a simplified 
newspaper article about catching a killer using a DNA profile of a blood sample from 
the crime scene. Students are asked to select the correct definition of DNA in a 
multiple choice format, and identify those questions about the crime which can be 
answered by scientific evidence. Another unit, MALARIA, provides a substantial text 
about the seriousness of the disease malaria, and about the life cycle of the malaria 
parasite in words and pictures. Students have to categorise methods of preventing the 
spread of malaria according to the stages in the life cycle of the parasite that are 
affected (OECD 2006, p. 30). 

A total of 85 mathematical literacy items were used in PISA 2003, with about half the 
items also included in the 2006 and 2009 PISA assessments, so that the trends in 
performance can be reported. The common items assessed in each cycle provide a link 
that enables the monitoring performance across and within countries over time. 
Ninety minutes of the assessment time were devoted to mathematical literacy in PISA 
2009, although not for any one student. In 2003 and again in 2012 when mathematics 
once again the major domain, the total assessment time for mathematics will be 270 
minutes. Many new items are in the preparation and field testing for the 2012 
assessment. 

Many of the PISA items are kept secure, so that they can be reused to create the trend 
data that is so valuable for governments monitoring progress in education systems. 
However, some items have been publicly released from every survey. Accessible from 
the PISA website and also downloadable as a booklet, PISA Take the test (OECD 
2009b) presents all the publicly available questions. Some of these questions were 
used in the PISA 2000, 2003 and 2006 surveys but will not be used again, and others 
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were used in trials but for some reason have not been included in the main survey. 
There are many reasons why an item might not be included in the final assessment 
and hence may be publicly released. For example, it may have not worked well in 
many or a few countries, it may have been too difficult or too easy, or there may have 
already been sufficient items with its characteristics so that it was surplus to 
requirements. 

PISA scores are reported along specific scales that are divided into levels, beginning 
at Level 1 with questions that require only the most basic skills to complete and 
increasing in difficulty with each level. In each test subject, the country score is the 
average of all student scores in that country. Some statistical adjustment is applied. 
The percentage of students at each level is also reported. Using Rasch measurement 
principles, the same scale is used to describe student ability and the difficulty level of 
each of the items. The scores have been arranged so that the average score among 
OECD countries is 500 points and the standard deviation is 100 points. As might be 
expected from economic factors, many of the non-OECD participating countries have 
lower scores. About two-thirds of students across OECD countries score between 400 
and 600 points. When the 2009 results were released (OECD 2010a), all observers 
were stunned when the municipality of Shanghai (China) obtained an average score in 
mathematics of 600, far out performing even the previous stand-out countries of 
Chinese Taipei (score 543), Finland (541) and Hong Kong-China (555). 

The continuum of increasing mathematical literacy from Level 1 to Level 6, with 
summary descriptions of the kinds of mathematical competencies at sample levels of 
proficiency, is shown in Table 1. A difference of about 60 score points represents one 
proficiency level, and 40 score points corresponds to about one year of schooling 
(OECD 2004, p. 60). 

Table 1. Description of mathematical literary of students at sample proficiency levels 
[From OECD 2009c, p. 122] 

Level 6 At Level 6, students can conceptualise, generalise, and utilise information 
(over based on their investigations and modelling of complex problem 

669.3 situations. They can link different information sources and representations 

score and flexibly translate among them. Students at this level are capable of 

points) advanced mathematical thinking and reasoning. These students can apply 
this insight and understandings along with a mastery of symbolic and 
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Level 4 

(over 

544.7 

score 

points) 


Level 2 

(over 

420.1 

score 

points) 


formal mathematical operations and relationships to develop new 
approaches and strategies for attacking novel situations. Students at this 
level can formulate and precisely communicate their actions and 
reflections regarding their findings, interpretations, arguments, and the 
appropriateness of these to the original situations. 

At Level 4, students can work effectively with explicit models for 
complex concrete situations that may involve constraints or call for 
making assumptions. They can select and integrate different 
representations, including symbolic, linking them directly to aspects of 
real-world situations. Students at this level can utilise well-developed 
skills and reason flexibly, with some insight, in these contexts. They can 
construct and communicate explanations and arguments based on their 
interpretations, arguments, and actions. 

At Level 2, students can interpret and recognise situations in contexts that 
require no more than direct inference. They can extract relevant 
information from a single source and make use of a single 
representational mode. Students at this level can employ basic algorithms, 
formulae, procedures, or conventions. They are capable of direct 
reasoning and making literal interpretations of the results. 


Information About PISA 

A great deal of information about PISA is publicly available. The publicly released 
items are available at PISA Take the test (OECD 2009b), and the main website 
provides most reports as free downloads (http://www.pisa.oecd.org/) . The ‘MyPISA’ 
site is the portal to see all the instruments used in the questionnaires, complete 
databases, published reports and also for a free data analysis service, where specific 
queries related to countries can be submitted for immediate response 
(http://mvpisa.acer.edu.au/) . 

The PISA Concept of Mathematical Literacy 

PISA (OECD 2006) defines mathematical literacy as an individual’s “capacity to 
identify and understand the role that mathematics plays in the world, to make well- 
founded judgments, and to engage in mathematics in ways that meet the needs of that 
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individual’s current and future life as a constructive, concerned and reflective 
citizen.” (p. 16). This definition will be refined and updated for the 2012 assessment. 
Whilst mathematical literacy is intended to highlight mathematical skills and 
understanding that are of use in future life, the intention is not to refer only to simple 
mathematics involved in straightforward activities such as shopping: it also 
encompasses preparation to use mathematics in technical professions of the highest 
level. 

The concept of mathematical literacy is closely related to several other concepts 
discussed in mathematics education. The most important is mathematical modelling 
(related to mathematisation by de Lange, 2006) and its component processes. These 
processes relate to formulating real world problems in mathematical terms so that they 
can be solved as mathematical problems, and then the mathematical solution can be 
interpreted to provide an answer to the real world problem. In the formulation stage, 
the problem solver faces a problem situated in a real context, and then gradually trims 
away aspects of reality, recognising underlying mathematical relations, and describes 
the stripped down problem in mathematical terms. In the interpretation stage, the 
problem solver considers the mathematical result(s), and uncovers their meaning in 
terms of the real context. Mathematical modelling has been an important concern 
amongst mathematics educators for many years (see, for example, Blum, Galbraith, 
Henn & Niss, 2007). Where mathematical modelling is seriously taught, students 
spend substantial time on one problem, moving through the whole cycle from 
formulating the problem mathematically, to solving it in mathematical terms and then 
interpreting and critiquing the solution. This critique might even demonstrate the need 
to begin again with a better formulated mathematical model. A key challenge for 
PISA is that it embraces the goals of mathematical modelling, yet it is not possible to 
ask substantial modelling questions under the constraints of timed assessment and the 
demand that results be amenable to standard statistical analysis. Only relatively short 
units that highlight some of these component skills can be used. 

Mathematical literacy is also associated with the concept of numeracy, which is 
especially prevalent in countries influenced by the British tradition. To give an 
example, in Australia the term numeracy is often used synonymously with 
mathematical literacy. In fact, though, different authors give it meanings ranging from 
a narrow ability with number to a very ambitious goal of using mathematics in many 
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different situations (Willis, 1990). In the United States, researchers concerned about 
the preparation of students for living have used terms such as quantitative literacy 
(Steen, 2001). PISA’s mathematical literacy, however, is not meant to imply a low- 
level basic competency. Instead, individuals in all walks of life need mathematical 
literacy, to different degrees in different occupations and life choices. 

Items for PISA mathematics are currently described on several dimensions. Attending 
to these dimensions ensures that the test is balanced against important criteria, as well 
as ensuring that the test remains true to the criteria for mathematical literacy. There 
are three main components in the framework that has been used up to 2009, although 
some changes are underway for 2012. One component is ‘overarching idea’ which 
classifies problems according to the content and purpose behind the use of 
mathematics, noting whether it deals with quantity, uncertainty, change and 
relationships or shape and space. These divisions are loose, but broadly align with the 
driving ideas behind the curriculum in most countries. The purpose of using 
‘overarching idea’ rather than the labels of syllabus topics is to highlight the nature of 
problems that mathematics contributes to solving. So for example, there are many 
syllabus topics that assist in solving problems related to the overarching idea of 
‘change and relationships’: derived quantities and units (e.g. speed, density), rates of 
change, ratio, calculus, algebraic functions). These syllabus topics could also be 
useful in solving problems related to the other overarching ideas. It is the underlying 
character of the problem, not the mathematical topic(s) involved that determines the 
‘overarching idea’ allocated to a problem. 

Another dimension is to classify the situation or context in which the items are 
embedded. These are classified (OECD 2006) as personal (related to everyday life), 
educational/ occupational (which might include contexts from other school subjects or 
the work of someone such as a bricklayer), public (e.g., a context relating to interest 
rates) and scientific (e.g. interpreting a graph about the growth of microorganisms). A 
key question for PISA, which stresses the usefulness of mathematics for life, is to 
what extent intra-mathematical items are allowed. Should every item have a clear 
reference to a situation beyond mathematics, or a simplification of one? Or can some 
items belong entirely to the mathematical world? Whereas TIMSS items test the 
curriculum (so solving an equation such as 9x + 6 = x + 14 is a reasonable question). 
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PISA items test knowledge in context. There are few PISA items with a context 
entirely within mathematics. 

Choice of PISA items in mathematics also raises issues of the interpretation of the 
‘educational context’. This is because mathematics is taught at school both for 
immediate use and as preparation for future study. Governments wish to know how 
well schools are preparing students in both these respects. It is not in the spirit of 
PISA to include an item only requiring students to solve a quadratic equation, for 
example, although such decontextualised mathematics is typical of the formal 
preparation for future study actually given in many countries and schools. As such, 
PISA might be seen as being out of step with mathematics as it is taught in some 
classrooms. For some, this is a criticism but is better viewed as an inspiration to shift 
mathematics teaching so that students are more likely to be able to use what they 
learn. 

PISA and Curriculum Content 

The intention of PISA is to measure how well students are prepared at school to use 
what they have learned to analyse and reason as they interpret and solve problems in a 
variety of situations encountered in life. Of course, in order to assess mathematical 
literacy of 15-year-olds, there is a need for items to be within the capacity of 15-year- 
olds in participating countries, and hence to correspond with school curricula in a 
broad sense. Items must also take into account the interests and knowledge of the 
world that 15-year-olds are likely to have encountered. Saying that the PISA 
assessments are not designed around school curricula does not imply that either the 
survey items or the results are not influenced by them. For example, Wu (2010) 
showed how countries’ national performance is affected by the alignment of its 
curriculum with the PISA assessment. Australia, for example, usually does well in the 
assessment of mathematical ideas of chance and the interpretation of data, and these 
topics are well represented in the Australian curriculum. Studies such as this reinforce 
the view that if scientific and mathematical literacy are regarded as valuable outcomes 
of schooling, then they should be well represented in intended and implemented 
curricula. 
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PISA Mathematics Scores 

As is to be expected from such enormous collection of data (approximately 470 000 
students in 2009), PISA produces many results. Rankings are always of interest, even 
though there are many complex factors that contribute to students’ proficiency. In this 
paper, results for Indonesia are compared with the OECD average and with the results 
for some neighbouring countries (Australia, Hong Kong-China, Japan and Thailand). 
Finland is also included because it has received much attention as a consistently very 
high-ranking OECD country in all three literacies. 

With the results from four cycles of PISA testing now available, trends over time are 
becoming increasingly valuable. Figure 1 shows the mean scores for mathematics, 
science and reading for Indonesia for each of the four PISA assessments. The standard 
deviations are between 66 and 85. There has been a steady increase in mean scores for 
the reading scale since 2000. The increase of 31 points from 2000 to 2009 is the 
fourth largest increase of any country for reading (OECD 2010a, Vol. V, Figure 
V.2.1, p. 39). The 2009 mean for science shows a drop of 10 points from a fairly 
stable level in the previous three assessments. A small decrease like this in a country’s 
score is not uncommon, but it is surprising when compared with the consistent gains 
made in reading in Indonesia. The mathematics score has been more unstable. One 
way of interpreting the data is that it has been steady, except for a relatively high 
score in 2006. The summary report for PISA 2006 noted that the high scores in 2006 
in both reading and mathematics were “largely driven by the higher performance of 
[Indonesian] males in PISA 2006” (OECD 2007a, p. 320). In terms of country 
rankings, these scores place Indonesia towards the end of the ranked list. Colombia, 
Albania, Tunisia, Qatar, Peru and Panama had an average score not statistically 
different from Indonesia’s. Panama’s score of 360 was the second lowest score for 
mathematics in 2009. 
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Figure 1. Mean scores of Indonesian students for mathematics, science and reading 
for PISA 2000 - 2009 [Data from OECD 2003a, Tables 2.3a, 3.1, 3.2; OECD 2004, 
Tables 2.5a, 6.2, 6.6; OECD 2007, Vol. 1, Tables 2.1c, 6.1c, 6.2c; OECD 2010a, Vol. 
I, Table 1.2.2, Vol. V. Tables V.2.9, V.3.3], 

Table 2 shows the percentages of students reaching the top two levels (Levels 5 and 6) 
for science, mathematics and reading in the PISA 2009 assessment. The PISA 2009 
report (OECD 2010a) states that students “who get to this level can be regarded as 
potential ‘world class’ knowledge workers of tomorrow, making the proportion of a 
country’s students reaching this level relevant for its future economic 
competitiveness” (Vol. I, p. 51). Indonesian 15-year-olds are not yet reaching this 
level. 

Table 2. PISA 2009: Percentage of students reaching the top two levels (Levels 5 and 
6) in PISA 2009 [From OECD 2010a, Vol. I, Tables 1.2.1,1.3.1,1.3.4] 


Country Percentage of students reaching the top two levels 



Science Literacy 

Mathematics 

Literacy 

Reading Literacy 

Indonesia 

0 

0.1 

0.02 

OECD average 

8.5 

12.7 

7.6 

Australia 

14.5 

16.4 

12.8 
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Finland 

18.7 

21.7 

14.5 

Hong Kong-China 

16.2 

30.7 

12.4 

Japan 

16.9 

20.1 

13.4 

Thailand 

0.6 

1.3 

0.3 


At the other end of the scale, the report notes that “Level 2 represents a baseline level 
of mathematics proficiency on the PISA scale at which students begin to demonstrate 
the kind of skills that enable them to use mathematics in ways that are considered 
fundamental for their future development” (OECD 2010a, Vol I, p. 132). A high 
proportion of Indonesian students are still below level 2 at age 15. Table 3 shows the 
percentages of students in selected countries scoring below Level 2 in PISA 2009 for 
the three literacies. 

On average, 22% of students in OECD countries perform below Level 2 in 
mathematics, but there are wide differences between countries. In 6 countries 
(including high performing Asian participating economies), less than 10% of students 
performed at below level 2. In all other OECD countries, the percentage of students 
performing below Level 2 ranges from 11.5% in Canada to 51.0% in Chile (OECD 
2010a, Vol. I, p. 133). As noted above, as evident in Table 3, non-OECD participating 
countries tend to have more students at the low levels of achievement. For this reason, 
the 2012 assessment will have options for countries to include more items aimed at 
this group of students, in order to give more sensitive assessments in this range. One 
benefit of this strategy should be to reduce the percentage of students who judge a 
question to be too difficult and hence do not respond. 

Table 3. Percentage of students below Level 2 in PISA 2009 [From OECD 2010a, 
Vol. I, Tables 1.2.1,1.3.1,1.3.4] 

Country Percentage of students below Level 2 

Science literacy Mathematics Reading literacy 

literacy 


Indonesia 

65.6 

76.7 

53.4 

OECD average 

18.0 

22.0 

18.8 

Australia 

12.6 

15.9 

14.2 

Finland 

6.0 

7.8 

8.1 

Hong Kong-China 

6.6 

8.8 

8.3 
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Japan 10.7 12.5 13.6 

Thailand 42.8 52.5 42.9 


Table 4 and Figure 2 show the distributions of performance for 2009 PISA 
mathematics. The distribution of Indonesian students across the levels resembles the 
distribution for Thailand, but Thailand has approximately 20% fewer of its students in 
Levels 1 and 2. Looked at another way, the scores of Indonesian students are about 50 
points below the Thai students across the distribution, and this is of the order of a full 
year of schooling difference. The mean OECD score is above the 95 th percentile for 
Indonesian students and between the 75 th and 90 th percentile for Thai students. 

Table 4. Student performance for PISA 2009 mathematics, showing percentile scores 
[From OECD 2010a, Vol. I, Table 1.3.3] 


Country Mean Percentiles 




5th 

10th 

25th 

75th 

90th 

95th 

Indonesia 

371 

260 

284 

324 

416 

462 

493 

OECD average 

496 

343 

376 

433 

560 

613 

643 

Australia 

514 

357 

392 

451 

580 

634 

665 

Finland 

541 

399 

431 

487 

599 

644 

669 

Hong Kong-China 

555 

390 

428 

492 

622 

673 

703 

Japan 

529 

370 

407 

468 

595 

648 

677 

Thailand 

419 

295 

321 

365 

469 

522 

554 
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Distribution of Mathematical Lite racy for Se lected 
Populations (from 5th to 95th percentile) 



Figure 2. Distribution of mathematical literacy score for 5 th , 10 th , 25 th , mean, 75 th , 90 th 
and 95 th percentiles [Data from OECD 2010a, Vol. I, Table 1.3.3]. 

Measures of Equity 

The 2009 report (OECD 2010a) notes that there is a correlation between GDP per 
capita and educational performance, but this only predicts 6% of the differences in 
average student performance across countries. The other 94% of difference reflects 
the fact that countries of similar prosperity can produce very different educational 
results (Vol. I, p. 161). In all countries, achievement of individuals is statistically 
related to various factors in the individuals’ socio-economic background, but in some 
countries this plays a greater role that in others. This can provide important 
information to countries about the equity of their school provision. 

Parental Education 

Parental education is an important predictor of achievement. PISA 2006 analysed 
student results according to parental education, as reported on the student 
questionnaire. Table 5 shows the mean scores for two groups of students: those with 
parents with the highest categories of education and those with parents with the lowest 
levels of education. Across all countries and across the three domains there are 
marked differences in mean scores between these two groups of students, regardless 





















The PISA View of Mathematical Literacy in Indonesia 


111 


of differences in economic, social and cultural status. In Indonesia the average 
differences between these two groups of students are comparable with those in the 
high-equity country of Finland (see OECD 2007, Vol. 1, Figure 4.10, p. 189) and are 
smaller than for Australia, Hong Kong-China or Thailand, and considerably smaller 
than the average differences across the OECD. The Indonesian difference of about 40 
points is approximately equivalent to 1 year of schooling. 


Table 5. Mean scores of students by parents’ level of education [From OECD 2007, 
Vol. 2, Table 4.7a, p. 135] 



Students whose parents have low level 

of education (lower secondary or less) 

Students whose parents have high level 

of education (tertiary) 


Science 

Mathematics 

Reading 

Science Mathematics 

Reading 

Indonesia 

378 

375 

377 

416 

416 

414 

OECD av. 

446 

448 

443 

525 

522 

516 

Australia 

486 

487 

477 

548 

539 

532 

Finland 

532 

517 

518 

572 

556 

556 

HK-China 

523 

529 

521 

575 

580 

561 

Thailand 

405 

400 

402 

482 

477 

475 


Socio-Economic Background 

In order to summarise the influence of socio-econo mi c background on the 
performances of students and countries, an index of economic, social and cultural 
status (ESCS) was created by combining variables such as the parents’ occupational 
status, education, wealth, home educational and cultural resources compiled from the 
questionnaire data. Table 6 shows the relationship between student performance in 
mathematics and the PISA ESCS index for PISA 2006. For the OECD average and for 
the countries shown here, the score point differences between the bottom and top 
quarters are all statistically significant. The score point differences are equivalent to 
between half a year and one year of schooling on average. 
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Table 6. PISA 2006: Mean scores in mathematics of students with the lowest and 
highest index of economic, social and cultural status (ESCS) [From OECD 2007, Vol. 
2, Table 4.11, p. 157] 


Country 

Students in bottom 

Students in top quarter 

Difference in mean 


quarter of ESCS index 

of ESCS index 

scores 

Indonesia 

366 

431 

65 

OECD av. 

455 

544 

89 

Australia 

482 

561 

79 

Finland 

519 

585 

66 

HK-China 

517 

579 

62 

Japan 

485 

556 

71 

Thailand 

391 

466 

75 


A key concept developed to analyse the PISA 2003 results is that of social gradient 
(also called socioeconomic gradient). This is effectively the number of score points of 
performance that is associated with one unit of improvement in socio-econo mi c 
conditions as measured by the ESCS index. In mathematics in PISA 2003 (OECD 
2004), most of the Asian region countries (Hong Kong-China, Japan, Korea and 
Macao-China, Indonesia and Thailand) had below OECD-average social gradient. 
This means that the impact of socio-economic factors on student performance is less 
than in many other countries and, at least on a superficial interpretation, points to 
equity of schooling. This was again found for PISA 2009 (OECD 2010a, Vol. II, 
Figure II.3.4, p. 59) (except for Japan). There are many challenges for education in 
this region and many improvements need to be made, but the relative equity is a good 
characteristic of Asian educational systems. Table 7 shows the social gradient (the 
score point difference for mathematics associated with one unit of ESCS) and the 
percentage of explained variance in students’ scores for the 2006 mathematics 
assessment and the 2009 reading assessment. For Indonesia and Thailand, the 
relationship between PISA score and ESCS index is distinctly curved, so that an 
increase of one unit in the ESCS index makes more difference to achievement at 
higher levels. 
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Table 7. Relationship between student performance in mathematics and ESCS index 
for PISA 2006 [From OECD 2007, Vol. 2, Table 4.4d, p. 129; OECD 2010a, Vol. II, 
Table II.3.2] 


Country 

Social gradient (increase in maths 

score for one unit increase in 

index) 

Percentage of variance in student 

performance explained by ESCS 

index 

2003 

Mathematics 

2009 Reading 

2003 

Mathematics 

2009 Reading 

Indonesia 

+24* 

+17 

10.1 

7.8 

OECD av. 

+38 

+38 

14.4 

14.0 

Australia 

+38 

+46 

11.5* 

12.7 

Finland 

+32* 

+31 

10.0* 

7.8 

HK-China 

+26* 

+17 

6.8* 

4.5 

Japan 

+40 

+40 

9.4* 

8.6 

Thailand 

+28* 

+22 

14.6 

13.3 


Gender Differences 

In PISA 2009 reading, girls outperformed boys in every participating country by an 
average, among OECD countries, of 39 PISA score points - equivalent to more than 
half a proficiency level or one year of schooling (OECD 2004, p. 60). In contrast, 
gender differences in mathematics generally favour boys, but Indonesia is one country 
where there has been very little or no difference. This is another equity measure where 
differences common in other countries are less in Indonesia. On average across OECD 
countries, boys outperformed girls in mathematics by 12 score points in PISA 2009. 
However, there is considerable variation between countries as shown in Table 8. In 
PISA 2009, Indonesia is one of only 11 countries where the difference is in favour of 
girls, although this was not the case in PISA 2006.1n PISA 2006, the averages for 
boys and for girls were both above other years, but boys outperformed girls by 17 
score points. In contrast with the reading results, in mathematics there are more boys 
(6.6%) than girls (3.4%) amongst the top performers in the OECD countries (OECD 
2010a, Vol. I, Table 1.3.8). For Indonesia the corresponding percentages are 0.1% and 
0.0%. The mean scores by gender for Indonesia over three cycles are shown in Figure 
3. 
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Table 8. Gender differences in scores for mathematics in PISA 2009 [From OECD 
2010a, Vol. I, Table 1.3.3, p. 224] 


Country 

Mean 

S.D. 

Mean score 

Mean score 

Score difference 

All 


Boys 

Girls 

(B-G) 

Indonesia 

371 

70 

371 

372 

-1 

OECD av. 

496 

92 

501 

490 

12 

Australia 

514 

94 

519 

509 

10 

Finland 

541 

82 

542 

539 

3 

HK-China 

555 

95 

561 

547 

14 

Japan 

529 

94 

534 

524 

10 

Thailand 

419 

79 

421 

417 

4 


Mean scores for Indonesian students from PISA 2003 to PISA 2009 


410 



2003 2006 2009 


□All students nBoys aGirls 

Figure 3. Mean mathematics scores of Indonesian boys and girls from PISA 2003 to 
PISA 2009 [Data from OECD 2010a, Vol. I, Table 1.3.3, p. 224; OECD 2007, Vol. 2, 
Table 6.2c, p. 230; OECD 2004, Table 2.5c, p. 356], 

School Climate 

Information about many aspects of school climate and the learning environments is 
available from the PISA 2009 student and school questionnaires. Two examples are 
given here. The first is from the student questionnaire, where students respond to 
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Likert items on their relationships with teachers. Some sample results are given in 
Table 9, for responses to the following five items: 

Item A I get along well with most of my teachers. 

Item B Most of my teachers are interested in my well-being. 

Item C Most of my teachers really listen to what I have to say. 

Item D If I need extra help, I will receive it from my teachers. 

Item E Most of my teachers treat me fairly. 

The questionnaire data show that Indonesian students report very favourable views of 
teacher-student relations, being above the OECD average on nearly all measures. The 
negative responses from Japanese students have been of concern to Japanese 
educators for some years. 

Table 9. Percentage of students responding ‘agree’ or ‘strongly agree’ to five items 
[From OECD 2010a, Vol. IV, Figure IV.4.1 p. 89] 


Country 

Item A 

Item B 

ItemC 

Item D 

Item E 

Indonesia 

93 

82 

63 

85 

91 

OECD average 

85 

66 

67 

79 

79 

Australia 

85 

78 

71 

84 

85 

Finland 

87 

49 

63 

84 

80 

Hong Kong 

89 

71 

67 

89 

82 

Japan 

73 

28 

63 

64 

74 

Thailand 

87 

77 

82 

83 

87 


The second example is from the school questionnaire. Principals respond to Likert 
items on many aspects of their schools, including giving their views on how student 
behaviour in the school affects learning. The data in Table 10 show that in comparison 
to the selected countries, Indonesian principals are comfortable with these aspects of 
student behaviour in their schools. 

Item G Student absenteeism 

Item H Disruption of classes by students 

Item I Students skipping classes 

Item J Students lacking respect for teachers 

Item K Student use of alcohol or illegal drugs 

Students intimidating or bullying other students 


Item L 
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Table 10. Percentage of students in schools whose principals reported that the 
phenomena hindered the learning of students ‘not at all’ or ‘very little’ [From OECD 
2010a, Vol. IV, Figure IV.4.4, p. 95] 


Country 

Item G 

Item H 

Item I 

Item J 

Item K 

Item F 

Indonesia 

67 

92 

83 

85 

98 

94 

OECD average 

52 

60 

67 

76 

91 

86 

Australia 

52 

69 

77 

77 

96 

81 

Finland 

27 

38 

57 

67 

96 

71 

Hong Kong-China 

83 

83 

90 

84 

98 

92 

Japan 

67 

91 

89 

76 

98 

93 

Thailand 

39 

72 

62 

87 

88 

92 


What Causes Success on PISA? 

Comparisons between countries reveal a complex set of factors that are associated 
with high scores: there is no magic recipe for success. Finland’s success in PISA has 
been consistent from the beginning. It has always scored very highly (and often best) 
in reading, mathematics and science. This success has resulted in a great deal of 
interest in its educational system. Finland’s success in PISA is often attributed to the 
emphasis that their education system places on teacher training to guarantee high- 
standard instruction. During a conference in March 2005, aimed at educating other 
OECD countries on the reasons for Finnish success in PISA, Arvo Jappinen, the top 
civil servant at the Ministry of Education, explained that the main reason for Finland’s 
results was “Teachers, teachers, and teachers!” (Jappinen, 2005). In a similar way, the 
earlier success of Singapore and Japan in TIMSS mathematics created a great deal of 
interest in their teaching methods amongst researchers, governments and teachers 
around the world. Japanese lesson study as a method of professional development has 
become popular and the sale of Singapore’s textbooks in the United States flourishes. 

It seems that each high performing country explains its success in international 
assessments in a different way: Singapore refers to careful structuring of school and 
curriculum. Japan highlights carefully constructed lessons and its culture of lesson 
study. Finland points to teacher quality. Hong Kong-China points to a combination of 
strong procedural work, depth in the treatment of mathematics and the Confucian 
heritage. Netherlands (one of the best European performers) often cites its textbooks 
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and curriculum. We can all learn something from all of these, but these varied voices 
also highlight the fact that success can be achieved in many different ways, and that 
the path to improvement will be different in different countries. 


Achievement Across Mathematics Content Areas 

As noted above, from 2000 to 2009, PISA results were reported according to four 
‘overarching ideas’ (OECD 2009c, pp. 83-122). Figure 4 shows the results for 
Indonesian boys and girls. The OECD average is about 500 for each overarching idea. 
Indonesian students performed better on the uncertainty items and less well on the 
change and relationships items. Indonesian boys tended to perform better than girls on 
space and shape items, whereas Indonesian girls performed slightly better than boys 
on uncertainty items. A further analysis at the question level by hierarchical cluster 
analysis showed that the patterns of success on items for Indonesian students was 
closest to the responses of students from Tunisia, Japan and Korea (OECD 2009a, p. 
123). In common with Mexico and Brazil, questions presented in a personal context 
were relatively easier for students in Indonesia than problems presented in other 
contexts (for example, occupational, scientific). (OECD 2009a, p. 136) Other than 
this, there was no systematic statistically significant effect of context of questions on 
student performance. 


PISA 2003: Mean scores for Indonesian students 
for four mathematics scales 



Space/shape Change/relationships Quantity Uncertainty 


□ All students ■ Boys □ Girls 


Figure 4. Mean scores on four mathematics scales for Indonesian students for PISA 
2003 [Data from OECD 2004, Tables 2.1c, 2.2c, 2.3c, 2.4c]. 
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A separate analysis grouped items into five traditional topics. This demonstrated that 
PISA 2003 Algebra and Measurement questions were significantly more difficult for 
students than Number, Geometry and Data. Table 11 shows the mean difficulty index 
for these topic groupings. Each overall country mean is set to 0, so this table shows 
the relative difficulty of topics within countries, but not between countries. Figure 5 
shows the means and standard deviations for the five traditional topics for Indonesian 
students. 

Table 11. PISA 2003: Relative mean difficulty of items in PISA 2003 by five 
traditional topics. (Higher score is greater difficulty for items) [From OECD 2009b, 
p. 236] 


Country 

Algebra 

Data 

Geometry 

Measurement 

Number 


(7 items) 

(26 items) 

(12 items) 

(8 items) 

(32 items) 


Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Mean (SD) 

Indonesia 

0.59 (1.82) 

-0.03 (1.15) 

-0.31 (1.12) 

1.01 (0.99) 

-0.24 (1.29) 

Australia 

0.94 (1.39) 

-0.35 (1.24) 

-0.15 (1.18) 

0.95 (0.94) 

-0.10(1.06) 

Finland 

1.03 (1.31) 

-0.28 (1.20) 

-0.08(1.13) 

1.04 (0.89) 

-0.23 (1.20) 

HK-China 

0.51 (1.28) 

-0.11 (0.99) 

-0.26 (1.25) 

0.82 (0.82) 

-0.13 (1.20) 

Japan 

0.69 (1.29) 

-0.03 (1.24) 

-0.40 (1.30) 

0.82 (0.84) 

-0.19(1.42) 

Thailand 

1.04(1.40) 

-0.16(1.26) 

-0.36(1.13) 

1.08 (1.22) 

-0.23 (1.24) 


PISA 2003. Indonesia: Index of difficulty {mean and 
standard deviation) for five traditional topics 
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Figure 5. Relative mean difficulty (and standard deviation) of five groups of items for 
Indonesian students in PISA 2003 [Data from OECD 2009a, p. 236]. 
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Sample PISA Items 

A complete set of results is available from the on-line data service accessed through 
the ‘MyPISA’ website and all publicly released items can be found in PISA: Take the 
test (OECD 2009b). In this paper, we include some comments on just two sample 
items. The descriptions of the questions and the data for the responses of Indonesian 
students are taken from Learning mathematics for life: A Perspective from PISA 
(OECD 2009a). 

MATHEMATICS UNIT 6: Growing up 

Growing up (Figure 6) presents students with graphs showing the relationships 
between age in years and average height in centimetres for young males and young 
females. Question 1 asks students to read a statement then to identify and carry out a 
simple calculation. Additional information from the graph is not required. Question 2 
asks students to compare the two line graphs to identify the ages when females are 
taller than males. Question 3 asks students to explain how the shape of the line graph 
reflects changes in the growth rate. 


GROWING UP 

Youth grows taller 
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Question 1: GROWING UP 

Since S9S0 t he average height of 20 rear old females has increased by 2.3 cm, 2d J7tf.fi cm. 

M'TwC was ihe average height of a 20 v ear old female in 19SO? 

A nswer; ........ r .cm 

Question 2: GROWING UP 

According to this graph, on average, during which period in their hfe ore females taller shun males oj the same age? 


Question 3: GROWING UP 

t^Jiirr? flow 2 fit: graph shows that on average the growth rate for girls slows down after 12 years of age. 


Figure 6. Unit: Growing up [From OECD 2009a, pp. 50, 52, 90]. 

Question 1 illustrates Level 2 in PISA 2003 and has a difficulty of 477 PISA score 
points. On average across OECD countries, 67% of students across the OECD and 
19% of Indonesian students were correct (OECD 2009a, Table 3.1 p. 47; p.51; 
(OECD 2009a, Figure A1.6). 

Question 2 illustrates two different levels of proficiency depending on whether 
students gave a fully or partially correct answer. Here, a partially correct answer 
scored illustrates exactly the boundary between Level 1 and Level 2 with a difficulty 
of 420 PISA score points. A fully correct answer illustrates Level 3 with a difficulty 
of 525 score points. The percentage of students giving partially or fully correct 
solutions was 69% (OECD) and 39% (Indonesia) whilst the percentage with fully 
correct answers was 28% (OECD) and 12% (Indonesia) (OECD 2009a, Table 3.2, p. 
63; Figure A1.7). 

For full credit, a response to question 3 should refer to the change of the gradient of 
the graph for females. This can be done by explicitly discussing the steepness of the 
curve of the graph or implicitly using a measure of numerical growth before and after 
12 years of age. Question 3 illustrates Level 4 in PISA 2003 mathematics with a 
difficulty of 574 PISA score points. In the OECD, 45% of students answered this 
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question correctly compared with 11% in Indonesia (OECD 2009a, Table 3.2, p. 63; 
Figure A 1.22). 

Figure 7 shows how Indonesian students responded to Growing up Questions 1, 2 and 
3. The percentages of students answering each question correctly follow the described 
difficulty levels, with students finding the Fevel 1/Level 2 question 2 easiest and the 
Level 4 question 3 most difficult. 


Question 1 
Question 2 
Question 3 


-100 -80 -60 -40 -20 0 20 40 60 80 100 

Percentage of students 

Gave incorrect answer Gave partially correct answer 

Did not answer the question Q Gave correct answer 

Did not reach the question 

Figure 7. How Indonesian students responded to Questions 1, 2 and 3 of the unit 
Growing up [Data from OECD 2009a, Figure A1.6, p. 205; Figure A. 1.7, p. 206; 
Figure A. 1.22, p. 221], 

MATHEMATICS Unit 13: Exports 

The unit Exports (Figure 8) presents students with a column graph and a pie chart. 
Students must decide which of the two graphs is relevant to each question and must 
locate the correct information in that graph. 
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EXPORTS 


The graphics below show lnforntai ion about expertsJrom Zedhnd. j country that uses zeds as Jfj currency. 


Tu t a. I Annual exports From Zed Land in 
mEUiomaf^ 199^-2000 


EM^tributbn of exports from 
£cdLand in 2CKHJ 



IMS LW IWS 1999 2DD0 


Year 


Meat 

t4% 



Question 1: EXPORTS 

What Mi -is the toiuf value fin mii lions of sods) of exportsfrom Zedland rn I99S? 

Answer:. 


Question 2: EXPORTS 

FiTitfj was the value cfjruh juice £j.parted from Z.ediand in 2000? 

A I .S million ^eds„ 

B 2.3 million zeds, 

C 2 A million zeds. 

D 3.4 million zeds. 

E 3.6 million xods. 


Figure 8. Unit: Exports [From OECD 2009a, pp. 169, 170]. 


Question 1 illustrates Level 2, with a difficulty of 427 PISA score points. This was 
answered correctly by 70% of Indonesian students and 79% across the OECD. 
(OECD 2009a, p. 55). This contrasts with the overall results for Indonesia, which 
showed that almost 70% of Indonesian students were below Level 2 over all topics. 
Question 2, which illustrates Level 4 and has difficulty of 565 PISA score points was 








































The PISA View of Mathematical Literacy in Indonesia 


123 


answered correctly by 48% of OECD students and 30% of Indonesian students 
(OECD 2009a, p. 86). Figure 9 shows the various responses by Indonesian students. 

Question 1 
Question 2 

-100 -80 -60 -40 -20 0 20 40 60 80 100 

Percentage of students 

| j Gave incorrect answer ^ Did not reach the question 

| | Did not answer the question Gave correct answer 



Figure 9. How Indonesian students responded to Questions 1 and 2 of the unit Exports 
[Data from OECD 2009a, Figure A1.3, p. 202; Figure A.1.20, p. 219]. 


The Future 

This paper has aimed to provide an introduction to the PISA program and to give the 
flavour of some of its findings. Through the numerous primary and secondary 
analyses of data that are now publicly available, PISA provides a very substantial 
body of information about education in mathematics. This paper has only been able to 
show a few examples. 

There are several innovations for PISA 2012 (OECD 2010b). First the definition of 
mathematical literacy is being revised to clarify the construct. An important 
refinement is to make it clearer that mathematical literacy is not intended to be a low 
level ability based on basic knowledge and skills, but something that individuals 
possess and require to various degrees. There will also be some simplification and 
clarification of the names of the various dimensions of the framework and a stronger 
indication of the mathematical content that might be useful in PISA items, although 
there is no intention to systematically assess any syllabus. 

The impact of computer technology on mathematical literacy is also receiving more 
attention, and as noted above, an optional computer-based assessment will be included 
in PISA 2012. The new PISA definition of mathematical literacy will recognise the 
important role of information technology by noting that more mathematically literate 
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individuals are better able to use mathematics and mathematical tools to make well- 
founded judgements and decisions. Working with electronic technology, whether 
packaged as calculators, computers or special purpose machines, is now an essential 
component of doing and using mathematics in developed countries and will 
increasingly impact on the lives of citizens in all countries through the time spans of 
the next cycles of PISA. The impact of computer technology on the ways in which 
individuals use mathematics, and consequently should learn it, has long been 
discussed, and continues to evolve. With the advent of four function calculators, the 
relative importance of methods of calculation for personal and occupational use 
changed. Pencil and paper algorithms diminished in importance, being gradually 
replaced by mental computation and estimation wherever appropriate and backed up 
by computer/calculator use. The use of calculators has been permitted in all PISA 
mathematics surveys, where this is consistent with the policy of the participating 
economy. The intention has not been to test calculator use of itself, or facility with 
any particular calculator capability, but to assist with calculation, as is normally done 
when individuals use mathematics. 

Additionally, we live increasingly in a society “awash in numbers” (Steen, 2001, p. 1) 
and “drenched in data” (Steen, 1999, p. 9) where “computers meticulously and 
relentlessly note details about the world around them and carefully record these 
details. As a result, they create data in increasing amounts every time a purchase is 
made, a poll is taken, a disease is diagnosed, or a satellite passes over a section of 
terrain.” (Orrill, 2001, p. xvi) After analysing the mathematical literacy required in 
industry and business to respond to the new data-rich environment, Hoyles, Wolf, 
Molyneux-Hodgson and Kent (2002, p. 3) noted that for employees at all levels in the 
workplace there is “an inter-dependency of mathematical literacy and the use of IT”. 
Full participation in society and in the workplace in this information-rich world 
therefore requires an extended mathematical literacy. The optional computer-based 
assessment in 2012 will be a first step towards assessing this. 

PISA is used by governments to monitor the performance of their educational 
systems. It can also describe characteristics of the educational system, including those 
which may impact on whether a society is harnessing the talents of all of its people. 
Through operationalising the construct of mathematical literacy, PISA has given voice 
to a vision of education that prepares all future citizens for living productive and 
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satisfying lives. It is up to mathematics education specialists in each country to 
promote curriculum, teaching and assessment that make this a reality. 
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